Skip to content

protocol v2.1: re-pin the engine, and make preview_play the engine again - #11

Merged
idIing merged 2 commits into
mainfrom
fix/engine-pin-and-preview-state-keys
Sep 6, 2026
Merged

idIing merged 2 commits into
mainfrom
fix/engine-pin-and-preview-state-keys

Conversation

@idIing

@idIing idIing commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Closes #9.

Two coupled changes that each move published numbers, landed together so the battery is
re-baselined once rather than twice, and shipped as protocol v2.1 — a point release: the
contract (items 1–7) is untouched, only the conditions under it moved.

1. Engine re-pin: 4d6f19dde733eb

The fork is rebased onto canonical Jackdaw 8712c1e, taking eleven upstream commits (including the
O(n²) get_x_same hoist and stake-sticker flag handling) and dropping two of its own that upstream
had absorbed.

The pin on its own moves nothing. Run with the new engine and the scorer untouched, the full
240-seed battery returns greedy-shop 3.204, random-shop 1.637, paired
+1.567 [+1.400, +1.729], 15,349 decisions — the v2.0 headline to the digit. Everything below is
therefore the scorer fix.

2. preview_play builds the scorer's game_state the way the engine does (#9)

GreedyTactical ranks every candidate play through preview_play, so this is inside the reference
agent's play selection, not a diagnostic path. #9 reported three keys score_hand reads that
preview_play never set. Auditing the class rather than the instance — intercepting score_hand on
both paths at real positions and diffing the two game_state dicts over the nineteen keys the
scorer actually reads — found four defects, at 99 of 99 audited positions:

key read at consumer preview was
hands_played scoring.py:447 Loyalty Card absent
skips scoring.py:458 Throwback absent
chips scoring.py:535,908 Mr. Bones absent
current_round_hands_played scoring.py:446 DNA, Sixth Sense +1 vs the engine

The fourth is not in #9. The engine increments both hand counters in the event queued after
evaluate_play (game.py:704-708), so every scoring context reads pre-increment values;
preview_play passed the post-increment one, which is exactly the value DNA and Sixth Sense test
against zero.

Measured effect before the fix, over six battery seeds: 218 of 36,941 previewed plays disagreed
with the engine's own dry-run of the same play — all 218 a Loyalty Card ×4 the engine applied and
the preview did not. After: 0 of 36,941, and 0 key disagreements.

The re-baselined battery

Full 240-seed train split, attributable (engine.commit de733eb, dirty false, no
"not attributable" warning), stamped jackhammer/v2.1:

v2.0 v2.1
greedy-shop mean highest ante 3.204 3.196
random-shop 1.637 1.637
random-legal 1.000 1.000
paired greedy-shoprandom-shop +1.567 [+1.400, +1.729] +1.558 [+1.396, +1.721]
greedy-shop decisions 15,349 15,307
random-shop / random-legal decisions 9,157 / 5,713 9,157 / 5,713
wins 0/240 0/240

Only greedy-shop moves. random-legal never runs the tactical layer; random-shop does, but its
recorded end-of-run inventories never hold any of the five jokers concerned (greedy-shop's do in
23 of 240 runs), and its per-seed outcomes and whole decision histogram are unchanged.

The regression gate

tests/test_exact_score.py, two forms because they fail independently:

  • class gate — diffs the engine's game_state against the preview's, key for key, at real
    positions. Catches a missing key and a present-but-wrong one.
  • outcome gate — preview total and hand type vs the engine's dry-run, over every play the
    tactical enumerates, on a window pinned to the board that actually diverged.

Both are red on the unfixed scorer and green after; 3.6s.

Everything in docs/known-limits.md was re-measured, not assumed

Every figure in that file was re-derived, not assumed. Almost all of them come back unchanged,
which is the useful result — the scan-cap audit is independent of the scorer:

  • paired shifts at --score-budget 8000: greedy-shop +0.104 [+0.046, +0.175], random-shop
    +0.004 [+0.000, +0.013]; difference-of-differences +0.100 [+0.046, +0.167]; 13/240 vs 1/240
    outcomes changed
  • 201 seeds where the cap never binds → dod exactly zero at zero variance; 39 exposed →
    +0.615 [+0.308, +0.974]
  • truncation: 576 scans in 30/240 games for greedy-shop, 151 in 14/240 for random-shop; max hand
    size 13; 0 truncation at 8000
  • Psychic lockups: 2 of 45 greedy-shop Psychic blinds, 0 of 26 for random-shop; 657P5QGW and
    PM4RVISW still 0/600 at ante 1 and still clear 720/600 at the raised cap
  • arms bit-identical through the ante-1 Small blind in 240/240; first divergence at a shop decision
    in 232/240, never a tactical one

What moved is only what the scorer touches: greedy's scan count 8341 → 8311 (6.91% → 6.93%
truncating), its decision histogram, the raised-cap plateau 3.308 → 3.300, and the sweep delta
+1.667 → +1.658. The 300→8000 cost was re-timed on this machine: 9.6% more combos for 9.6% more
wall clock (63.6s → 69.7s, 14 workers).

Why a point release and not v3

CONTRIBUTING.md requires a new protocol version for an engine-pin or evaluation-procedure change,
and docs/protocol-v2.md forbade editing a numbered item in place. Both are satisfied by v2.1, and
the distinction is now written into the protocol and into provenance.PROTOCOL: a major bump
means a clause changed meaning, so the two versions answer different questions; a point bump
means the contract is untouched and the conditions under it moved, so every figure is re-baselined
while still answering the same question. Without that rule a routine pin move would ship v3, v4, v5.

Checks

uv sync --locked                       clean
uv run ruff check src scripts tests    All checks passed
uv run ruff format --check ...         28 files already formatted
uv run pytest                          98 passed
scripts/evaluate.py --limit 1 -w 1     exit 0  (the CI smoke)
full battery, train, 240 seeds         3 agents, 0 fails, 0 fallbacks,
                                       protocol jackhammer/v2.1,
                                       engine de733eb source=installed-vcs dirty=false,
                                       kit 1.1.0, no attributability warning

The headline reproduced identically across four independent full-battery runs.

Note for the merge

pyproject.toml, CITATION.cff and the README install snippet move to 1.1.0; the v1.1.0 tag
has to be created at merge for that snippet to resolve.

idle and others added 2 commits September 5, 2026 21:14
Two changes that each move published numbers, landed together so the battery is
re-baselined once. Shipped as a point release: the contract (protocol items 1-7)
is untouched; only the conditions under it moved.

Engine pin 4d6f19d -> de733eb. The fork is rebased onto canonical Jackdaw
8712c1e, taking eleven upstream commits and dropping two it had absorbed. On its
own the pin moves nothing: with the scorer untouched the battery returns
greedy-shop 3.204, random-shop 1.637, +1.567 [+1.400, +1.729], 15,349 decisions
-- the v2.0 headline to the digit. Everything that moves is the second change.

preview_play now builds the scorer's game_state the way the engine does. It
reconstructs by hand the dict _handle_play_hand passes to score_hand, and
GreedyTactical ranks every candidate play through it, so it sits inside the
reference agent's play selection. #9 reported three keys it never set --
hands_played, skips, chips. Diffing the engine's game_state against the
preview's, key for key, at real positions found a fourth at 99 of 99 of them:
current_round_hands_played was passed one ahead, because the engine increments
both hand counters in the event queued after evaluate_play, which is exactly the
value DNA and Sixth Sense test against zero. A missing key and a stale key fail
identically -- silently, by defaulting -- so only the missing ones showed up in a
set difference.

Measured before the fix over six battery seeds: 218 of 36,941 previewed plays
disagreed with the engine's own dry-run, every one a Loyalty Card x4 the engine
applied and the preview did not. After: 0. tests/test_exact_score.py is the
standing gate, in two forms that fail independently -- a key-for-key diff of the
two game_state dicts, and preview total vs engine dry-run over every play the
tactical enumerates, on a window pinned to the board that diverged. Both are red
on the unfixed scorer.

Re-baselined battery, 240-seed train split, attributable (de733eb, not dirty):
greedy-shop 3.204 -> 3.196, random-shop 1.637 and random-legal 1.000 unchanged,
paired +1.567 [+1.400, +1.729] -> +1.558 [+1.396, +1.721], greedy-shop decisions
15,349 -> 15,307, wins 0/240 throughout. Only greedy-shop moves: random-legal
never runs the tactical layer, and random-shop's recorded inventories never hold
any of the five jokers concerned, where greedy-shop's do in 23 of 240 runs.

Every other figure in docs/known-limits.md was re-measured rather than assumed.
The scan-cap audit re-derives unchanged -- +0.104 [+0.046, +0.175] and +0.004
[+0.000, +0.013], difference-of-differences +0.100 [+0.046, +0.167], 13/240
against 1/240, the 201/39 split with dod exactly zero at zero variance and +0.615
[+0.308, +0.974] on the exposed seeds, 576 truncated scans in 30/240 games
against 151 in 14/240, max hand size 13, and the two Psychic lockups. What moved
is what the scorer touches: greedy's scan count 8341 -> 8311, its decision
histogram, and the raised-cap plateau 3.308 -> 3.300.

Closes #9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h gap

Adversarial review of the PR found four things. The fix itself and every
re-baselined number held; what did not were three claims in my own prose and one
coverage gap in the tests.

The docstring claimed preview_play now hands the engine what the engine would
hold, full stop. True of the scorer's own inputs, false as stated: synth is a
partial mirror and engine code other than score_hand runs against it. Three
reproducible Hook divergences -- money on a Mail-In Rebate board, a missing
discard-time suit target for Castle, and joker_slots copied pre-press_play so a
destroyed negative Ramen does not shrink it. All three predate the published v1
numbers and none is reached by the frozen battery, so they are logged as a known
limit and a contribution surface, not fixed here.

The versioning note said neither change touches a clause. It does: v2.1 edits
item 1's commit and changes how greedy-shop plays. The document now says plainly
that it amends the rule frozen at v2.0 -- which said any numbered item creates v3
-- splitting the rule by what moved rather than by whether a clause was edited,
and names that amendment as this release's only change to the contract's text.

"Both upstream commits read as exact-output" was scoped wrong. The stake-sticker
commit is a real behaviour change at Gold Stake; it is a no-op at item 3's White
Stake. The claim now attaches to the published configuration, not to the commits.

The 23/240 joker figure was a union of two partial sources and matched neither.
Replaying the streams: greedy-shop holds one of the five while playing a hand in
25 of 240 runs, random-shop in none -- and "dies too early to buy them" was
false, it acquires one in 5 runs and sells each before a play decision.

Coverage: every differential test would still pass if preview_play hard-coded
skips to 0, because no baseline skips a blind and the battery never produces a
nonzero value. Added a test that puts sentinels the battery cannot produce into
the live state and asserts they reach the scorer; it fails on the hard-coded
constant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@idIing

idIing commented Sep 6, 2026

Copy link
Copy Markdown
Owner Author

Adversarial review, and four corrections

Put to a second model cold — isolated clone of the pushed branch, the six claims handed over as
things to attack rather than as context, no framing of any of it as verified.

What held. The four key assignments; the pre-increment reading, cross-checked against the Lua
event order; the completeness of the nineteen-key audit at this pin; the gates are not vacuous
(reverting the fix gives 3 failed, 2 passed); and every re-baselined number, including an
independent 240-seed pin-only control that matched the old-pin events, summaries and blinds on all
240 seeds for both shop arms.

What did not. Three claims in the prose and one coverage gap — all in the write-up, none in the
fix. Fixed in 94d23a9:

  1. The exactness claim was universal and is not true. preview_play gets the scorer's inputs
    right, but synth is a partial mirror and engine code other than score_hand runs against it.
    Three reproducible Hook divergences: money on a Mail-In Rebate board, a missing discard-time
    suit target for Castle, and joker_slots copied pre-press_play so a destroyed negative Ramen
    does not shrink it. All predate the published v1 numbers and none is reachable from the frozen
    battery — which is precisely why the differential gates miss them. Docstring narrowed, logged in
    known-limits.md, filed as preview_play's synthetic game_state is too thin for the Hook discard path — three reproducible divergences #12. Not fixed here: the root cause is key-by-key reconstruction, and
    the durable fix wants its own before/after over the battery.
  2. "Neither change touches a clause" was false. v2.1 edits item 1's commit and changes how
    greedy-shop plays. The protocol now says openly that it amends the rule frozen at v2.0 —
    which said any numbered item creates v3 — splitting it by what moved rather than by whether a
    clause was edited, and naming that amendment as this release's only change to the contract's text.
  3. "Both upstream commits read as exact-output" was scoped wrong. The stake-sticker commit is a
    real behaviour change at Gold Stake (eternal/rental flip on a controlled roll); it is a no-op
    at item 3's White Stake. The claim now attaches to the published configuration, not the commits.
  4. The 23/240 joker figure was wrong — a union of two partial sources that matched neither.
    Replaying the streams: greedy-shop holds one of the five while playing a hand in 25 of 240
    runs, random-shop in none. And "dies too early to buy them" was false: it acquires one in 5
    runs (K1U9J9UF, Q358C3MG, K9ADQ6YV Mr. Bones; EZG9JGQS, JBVSIWHI Loyalty Card) and
    sells each before a play decision. The sharper claim was available all along.

Coverage gap, closed. Every differential test still passed with skips hard-coded to 0,
because no baseline skips a blind so the battery never produces a nonzero value. Added a test that
puts sentinels the battery cannot produce into the live state and asserts they reach the scorer; it
fails on the constant.

No published number changed. 99 passed, ruff clean.

@idIing
idIing merged commit 164b080 into main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

preview_play omits three game_state keys that score_hand reads — Loyalty Card's cycle freezes

1 participant